#7. 



o 
I- 




367 .40118X00 



IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 



o 



Applicant (s) : 



HANNUKSELA, et al . 



Serial No . : 



Filed: 



May 15, 2001 



Title : 



VIDEO CODING 



Group : 



LETTER CLAIMING RIGHT OF PRIORITY 



Honorable Commissioner of 



May 15, 2001 



Patents and Trademarks 
Washington, D.C. 20231 

Sir: 

Under the provisions of 3 5 USC 119 and 3 7 CFR 1.55, the 
applicant (s) hereby claim(s) the right of priority based on 
United Kingdom Patent Application No,(s) 0011639.2, filed 
May 15, 2000. 

A certified copy of said United Kingdom Application is 
attached . 



Respectfully submitted. 



ANTONELLI, TERRY, STOUT & KRAUS, LLP 




Carl I . Brundidge 
Registration No. 29,621 



CIB/alb 
Attachment 
(703) 312-6600 



THIS PAGE BLANK (uspto) 




The % 



Fktent 
I QE&ce I 



CERTIFIED COPY OF 
PRIORIIY DOCUMENT 




INVESTOR IN PEOPLE 



The Patent Office 
Concept House 
Cardiff Road 
Newport 
South Wales 
NPIO 8QQ 



o 



<D 

o 



I the undersigned, being an officer duly authorised in accordance with Section 74(1) and (4) 
of die Deregulation and Contracting Out Act 1994, to sign and issue certificates on behalf of 
die Comptroller-General, hereby certify that annexed hereto is a true copy of the documents 
as originally filed in connection with the patent application identified therein togedier with the 
Statement of inventorship and of right to grant of a Patent (Form 7/77), which was 
subsequently filed. 



In accordance with the Patents (Companies Re-registration) Rules 1982, if a company named 
in tiiis certificate and any accompanying documents has re-registered under die Companies Act 
1980 with the same name as that with which it was registered immediately before re- 
registration save for the substitution as, or inclusion as, the last part of the name of die words 
"public limited company" or dieir equivalents in Welsh, references to die name of die company 
in tiiis certificate and any accompanying documents shall be treated as references to die name 
with which it is so re-registered. 

In accordance widi die rules, die words "public limited company" may be replaced by p.l.c, 
pic, P.L.C. or PLC. 

Re-registration under die Companies Act does not constitute a new legal entity but merely 
subjects die company to certain additional company law rules. 




Signed 



VA 



Dated 10^^^ 



An Executive Agency of the Department of Trade and Industry 




I 



nts Form 7/77 



Patents Att 1977 
(Rule 15) 



Office 



Statement of inventorship and of 
right to grant of a patent 



lilt- ( f L. ! \j t :ijy -.f:- 



1 9 APR 2001: 



NEWPORT 



The Patent OfQce 

Cardiff Road 
Newport 
Gwent NP9 IRH 



1. 


Your reference 


PAT 00405 GB 


2. 


Patent application number 

(if you know it) 


0011639.2 


3. 


Full name of the or of each applicant 


NOKIA MOBILE PHONES LIMITED 
KEILALAHDENTIE 4 
02150 ESPOO 

FTNUiJMD ..- - 


4. 


Title of the invention 


VIDEO CODING 



5. State how the appiicantft> derived the right 

from the inventorf^ to be granted a patent BY VIRTUE OF AN ASSIGNMENT DATED lA MAY 2000 



How many, if any, additional Patents Forms 
7/77 are attached to this form? 

(see note (c)) 



7. 



I/We believe that the person^s; named over the page (and on 
any extra copies ofthisfom) is/arc the inventor^) of the invention 
which the above patent application relates to. 





nBAPF. HTRRFRT 



Date 



8. Name and daytime telephone number of 
person to contact in the United Kingdom 



JULIET HTBBERT 01252 865101 




Notes 

a) If y oil need help to fill in this form or you have any questions, please contact the Patent Office on 0645 500505- 
Write your answers i7i capital letters using black ink or you may type them. 

If there are more than three inventors, please write the names and addresses of the other inventors on the back of 
another Patents Form, 7/77 and attach it to this form. 

When an application does not declare any priority, or declares priority from an earlier UK application, you must 
provide enough copies of this form so that the Patent Office can se7id one to each inventor who is not an 
\pplica.nt 

\x:e you have filled in the form you must remember to sign and date it. 



Patents Form 7/77 



if 



Enter the full names, addresses and postcodes of the 
inventors in the boxes and underline the surnames 



MISKA HANNUKSELA 
KUKKANIITYNKATU 4 B 
33710 TAMPERE 
FINIvWD 



Patents ADP number (i/you know it): ? ~| Q^Oj ^^(^Q | 



KpREM CAGLAR 






LAHTEENKATU 5 H 116 






33500 TAMPERE 






FINLAND 






Patents ADP number a/ you know tt): 







Reminder 



Have you signed the form? 



Patents ADP number (if you know it): 



15-05-00 18:25 



+44 1252 065080 



P-02 



R-184 



Job-M3 



5-M(i^^00[MONl 18:25 
PaLuts Fonn 1/77 



Patents Act 1977 

CRole 16) 



NMP PATENTS UK 



FAX:t44 1252 865080 



P. 002 



Request for grant of a patent 

(Sec the notes aathxbAckcftbhfaTxa. Yao am »ii»o gd mo 



1 . Your reference 





Tbe Fafent Office 

Cardiff Road 
Newport 
Gwent NP9 IRH 



PAT 00iO5 GB 16HAY0O E536971-1 002716., 



201/7700 0 00-001 1;^>3? ^ 



2. Patent application number 

CTbG Patent Office wmf0 in this pari) 



1 5 MAY ZUUU 



0011639.2 



3. Full nanie , address and postcode of the Or of 
each appUcaDt Ctm^erUtte aU sumartuts) 



Patents ADP xmmber Of you know tt) 

If the applicant is a corporate body, give the 
country/state of Its incorporudon 



1K3KIA ^DBILE PHONES LIMITED 

(EUALAHDENTIE 4 

32150 ESPOO 
: FINLAND 



rTNLftND 



4. Title of the invention 



/IDEO CODING 



5 . NaxD« of your asent C^ycu have onO 

''Address for service*' in Che United Kingdom 
to which all correspondence should be sent 
(inctudfng tbmpGS$cotUy 



SfOKIA IPR DEPARTMENT 
SOKIA HOUSE 
SUMMIT AVENUE 
FARNBOROUGH 
aAMPSHIRE 
3U140NG UK 



Patents ADP number CiTyou knofw io 



6. If you are declaring priority from one or ipore 
earlier patent applications, the country 
and the date of filing of the or of each of these 
earlier applicatiom and Cfym Smow uj the or 
each application number 



Counuy 



7. If this application is divided or otherwise 
derived from an earlier UK application, 
give the number and the flUng date of 
the earUer application 



Number of carUc af^Ecalioo. 



8. Is a statement of inventorship and of right 
to grant of a patent required in support of 
this request? t^lfuuwr 749^ ^ 

a) at^ appHoant named inpart3 ^ inventor, or 

6l) tbeera is an inventor wbo is not nametX as an 
44ppiifxtntf or 

(tw^ narwtmd appHeani is a corporate hody, 
ttoio (4X> 



YES 



Priority s^pZicaUoa jxumber 
(^you know it} 



(day / month /year) 



Date offlUos 
(tiay / month /yea r) 



Patents Form 1/77 



15-05-00 18:25 



+44 1252 885080 



P. 03 



R-1B4 



Job-143 



S-MA-^'. ' OO(MON) 18:25 NMP PATENTS UK 
Pal Form 1/77 

9. Enter the nvimher of sheets for any of the 
following items you arc filing with this form. 
Do not count copies of the same document 

Continuation sheets of this form 

Description 

Claimr^^ 
Abstract 

Drawing^f> 



FAX:+44 1252 865080 



P. 003 



10. If you are also filing any of the following, 
state how many against each item. 

Priority documents 

Translations of priority docviments 

Statement of inventorship and riglU 
to grant of a patent (Fatents iform 7/77) 

lEtequest for preliminary examination 
and search fpatetits Form D/77) 

Request for substandve examination 

(PattmtS form 10/77} 

Any other docimients 



11. 




12. Name and daytime telephone number of 
person to contact in the United Kingdom 



grant of a patent on the basis of this application. 



lET HIBBERT 



ace 

12.05.00 



Miss J Hibbert 01252 865101 



After on application for a patent has beenfiied, the d^ptroUer of the Patent Office u/iil consider ti/hether puhOcation 
or comtnunication of the ini/ention should he prohibited or restricted under Section 22 of the Patents Act 1977, You 
will he informed if it is necessary to prohibit or restrict your inm ntion in this way. Furthermore, if you live in the 
United Kingdom, Section 23 of the Patents Act 1977 stops you frt m appfyingfor a patent abroad without first getting 
written permission from the Patent Office unless an appUcation i ias been filed at least 6 weefss heforeband in the 
Unued Kingdom for a putentfor the same invention and either no direction prohibiting publication or 
communication has been given, or any such direction has been i 'evoked 

Notes 

a) Jfyou need belp to fill in tbisform or you t*aw any questions, please contact the Patent Office on 0645 500505. 
by Write your answers in capital letters using black ink or you : nay type them. 

cj If there is not enough space for all the relevant details on any part of this form, please continue on u separate 
sheet of paper and write 'see continuation sheet** in the relet'ont partes}. Any continuation sheet should be 
attached to this fomu 

<0 V have answered 'Yes* Patents Porm 7/77 toiU need to bf filed. 

ej Once you have filled in the form you mzist remember to sigt, 



f} Poir details of the fee and ways to pay please contact the Pattrit Office. 



and date it. 



Patents Foixn 1/77 



15-05-00 18:25 



HA 1 252 865080 



P. 05 



R-184 



J Ob- 143 



5-MA^^0O(MON) 18:26 ' NMP PATENTS UK 



FAX:+44 1252 865080 



P. 005 



10 



15 



20 



25 



nc297eaa.doc 



VinEQ CODING 



PAT 00405 



This invention relates to video coding an|j in particular to concealing artefacts 
5 introduced by errors. 



30 



A video sequence consists of a seriesj of still pictures or frames. Video 
compression methods are based on reducing the redundant and perceptually 
in-elevant parts of video sequences. The redundancy in video sequences can 
be categorized into spectral, spatial and temporal redundancy. Spectral 
redundancy refers to the similarity between the different colour components of 
the same picture. Spatial redundancy results from the similarity between 
neighbouring pixels in a picture. Temporil redundancy exists because objects 
appearing in a previous image are also Ikely to appear in the current image. 
Compression can be achieved by taking advantage of this temporal 
redundancy and predicting the current picture from another picture, ternied an 
anchor or reference picture. Further compression is achieved by generating 
motion compensation data that describes the motion between the current 
picture and the reference picture. 



However, sufficient compression cannot 



usually be achieved by only reducing 



the inherent redundancy of the sequence. Thus, video encoders also try to 
reduce the quality of those parts of the x-ideo sequence which are subjectively 
less important. In addition, the redundancy of the encoded bit-stream is 
reduced by means of efficient lossless cading of compression parameters and 
coefficients. The main technique is to use variable length codes. 



these 



Video compression methods typically 
temporal redundancy reduction and 
that do not utilise temporal redundancy 
INTRA or l-frames or l-pictures. 
forwardly predicted from a picture occurring 



differentiate between pictures that utilise 
that do not. Compressed pictures 
reduction methods are usually called 
Terr^porally predicted Images are usually 
before the current picture and are 
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called INTER or P-frames. In the INTEFjl frame case, the predicted motion- 
compensated picture is rarely precise enough and therefore a spatially 
compressed prediction error frame is associated with each INTER frame. 
INTER pictures may contain iNTRA-coded areas. 

Many video compression schemes also use temporally bi-directionally 
predicted frames, which are commonly njferred to as B-pictures or B-frames. 
B-pictures are inserted between anchor picture pairs of I- and/or P-frames and 
are predicted from either one or both of these anchor pictures. B-pictures 
normally yield increased compression as compared with forward-predicted 
pictures. B-pictures are not used as an<:hor pictures, i.e., other pictures are 
not predicted from them. Therefore they can be discarded (intentionally or 
unintentionally) without impacting the picture quality of future pictures. Whilst 
B-pictures may improve compression performance as compared with P- 
pictures. their generation requires greater computational complexity and 
memory usage, and they introduce additional delays. This may not be a 
problem for non-real time encoding suc i as video streaming but may cause 
problems in real-time applications such as video-conferencing. 

A compressed video clip typically consists of a sequence of pictures, which 
can be roughly categorized into temporally independent INTRA pictures and 
temporally differentially coded INTER pictures. Since the compression 
efficiency in INTRA pictures is normally lower than in INTER pictures, INTRA 



pictures are used sparingly, especially in 



low bit-rate applications. 



A video sequence may consist of a number of scenes or shots. The picture 
contents may be remarkably differen from one scene to another, and 
therefore the first picture of a scene s typically INTRA-coded, There are 
frequent scene changes in television and film material, whereas scene cuts 



30 are relatively rare in video conferenci 



ig. In addition, INTRA pictures are 



typically inserted to stop temporal proDagation of transmission errors in a 



P, 006 



15-05-00 18:25 



*U 1252 865080 



P. 07 



R-1B4 



J Ob- 143 



15-MA^".00(MON) 18:27' NMP PATENTS UK 



( 



FAX:H4 1252 865080 



P. 007 



10 



20 



25 



30 



nc2978aa.dDC 

3 

reconstructed video signal and to providq random access points to a Video bit- 
streann. 

Compressed video is easily corrupted b/ transmission errors, mainly for two 
reasons. Firstly, due to utilisation of temporal predictive differential coding 
(INTER frames), an error is propagated both spatially and temporally. In 
practice this means that, once an error occurs, it is easily visible to the human 
eye for a relatively long time. EspeciallJ susceptible are transmissions at low 
bit-rates where there are only a few IN T RA-coded frames, so temporal error 
propagation is not stopped for some tme. Secondly, the use of variable 
length codes increases the susceptibility to errors. When a bit error alters the 
codeword, the decoder will lose codeword synchronisation and also decode 



subsequent error-free codewords (comp 
next synchronisation (or start) code. A 



1sing several bits) incorrectly until the 
synchronisation code is a bit pattern 



15 which cannot be generated from any Uigal combination of other codewords 



t stream at intervals to enable re- 
occur when data is lost during 



and such codes are added to the b 
synchronisation. In addition, errors 
transmission. For example, in video applications using the unreliable UDP 
transport protocol in IP networks, netwc rk elements may discard parts of the 
encoded video bit-stream. 

There are many ways for the receiver to address the corruption introduced in 
the transmission path. In general, on raceipt of a signal, transmission errors 



are first detected and then corrected 



or concealed by the receiver. Error 



correction refers to the process of recovering the erroneous data perfectly as 
if no errors had been introduced in the first place. Error concealment refers to 
the process of concealing the effects of transmission errors so that they are 
hardly visible in the reconstructed video sequence. Typically some amount of 
redundancy is added by the source or ransport coding in order to help error 
detection, correction and concealment- Error concealment techniques can be 
roughly classified into three categoric ts: forward error concealment, error 
concealment by post-processing and interactive error concealment. The term 
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those techniques in which the 
ansmitted data to enhance the error 



"forward error concealrnent" refers tc 
transmitter side adds redundancy to the t 
resilience of the encoded data. Error concealment by post-processing refers 
to operations at the decoder in response to characteristics of the received 
signals. These methods estimate the correct representation of erroneously 
received data. In interactive error concealment, the transmitter and receiver 
co-operate in order to minimize the efect of transmission errors. These 
methods heavily utilise feedback information provided by the receiver. Error 
concealment by post-processing can a 
concealment whereas the other two cate gories represent forms of active error 
concealment. 



There are numerous known concealment 
given by Y. Wang and Q. -F. Zhu in "Error 
Communication: A Review". Proceedings 
1998, pp. 974 - 997 and an article by P. 
"Error Concealment in Encoded Videlo 
Selected Areas in Communications. 



so be referred to as passive error 



algorithms, a review of which is 
Control and Concealment for Video 
of the IEEE, Vol. 86. No, 5, May 
Salama. N. B. Shroff, and E. J. Delp. 
submitted to IEEE Joumal on 



Current video coding standards define d syntax for a self-sufficient video bit- 
stream. The most popular standards at the time of writing are ITU-T 
Recommendation H.263, "Video coding for low bit rate communication", 
February 1998; ISO/IEC 14496-2, "Generic Coding of Audio-Visual Objects. 
Part 2: Visual". 1999 (known as MPEG--'! ); and ITU-T Recommendation H.262 
(ISO/IEC 13818-2) (known as MPEG-2). These standards define a hierarchy 
for bit-streams and correspondingly for image sequences and images. 



mpe:g 



To assist in error concealment, the 
the transmission of motion vectors fo 
pictures. These motion vectors are 
follows: if an INTRA macrobiock is lost 
motion vectors belonging to the 



-2 video coding standard allows for 
r INTRA macroblocks within INTRA 
ulsed only for error concealment, as 
(or corrupted), the decoder uses the 
madroblock above the lost one to get 
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resembling blocks from a reference picti 
contain motion information, the decodi 
algorithm. 



lire. If the INTRA macroblock does not 
r conceals the en"ors with a spatial 



In H.263, the syntax has a hierarchic 
picture segment, macroblock, and block 
parameters affecting the whole picture 
data. Most of this data is arranged in a 



I structure with four layers: picture, 
layer. The picture layer data contain 
area and the decoding of the picture 
-called picture header. 



so 



The picture segment layer can either be a group of blocks layer or a slice 
layer. By default, each picture is dividend into groups of blocks. A group of 
blocks (GOB) typically comprises 16 successive pixel lines. Data for each 
GOB consists of an optional GOB headnr followed by data for macroblocks. If 
the optional slice structured mode is used, each picture is divided into slices 
instead of GOBs. A slice contains a njmber of successive macroblocks in 
scan-order. Data for each slice consists 



the macroblocks. 



Each GOB or slice is divided into macroblocks. A macroblock relates to 16 x 



20 16 pixels (or 2 x 2 blocks) of luminance 



of a slice header followed by data for 



and the spatially corresponding 8x8 



pixels (or block) of chrominance comporjents. A block relates to 8 x 8 pixels of 
luminance or chrominance. 



Block layer data consist of uniformly 
coefficients, which are scanned in zigzag 
encoder and coded with variable lengt 
hierarchies resemble that used in H.263 



quantised discrete cosine transform 
order, processed with a run-length 
codes. MPEG-2 and MPEG-4 layer 



30 



In H.263, the issue of error concealn^ent is typically perceived as a post- 
processing function and is generally left to the decoder. In ITU-T Study Group 
16 Question 15 documents no. 17, 18, - 9, 20, 21 & 22, presented at the Ninth 
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meeting of ITU-T Study Group 16 in Ne\w 



Jersey in the USA in October 1999, 



it is proposed to add normative languacfo to H.263 to specify several error 
concealment techniques and to define e. signalling mechanism by which an 
encoder can annoiince this to a decode r. preferably on a picture-by-picture 
basis. 



However this approach is unduly restricjtive on the decoder since the error 
concealment method to be used by the 
Thus other concealment methods canncbt 
these methods available to use. 



decoder is specified by the encoder, 
be usedp even if the decoder has 



According to a first aspect of the invention there is provided a method of 
encoding a video signal representing £ sequence of pictures, the method 
comprising comparing a first picture with a second picture, calculating a 
measure of the similarity between the first and the second pictures, comparing 
the measure of similarity with a predetermined criterion of similarity and, when 
the measure of similarity does not meet the predetermined criterion of 
similarity, outputting an indicator indica ing that a non-temporally predictive 
error concealment method should be i sed by a subsequent decoder and, 
when the measure of similarity meets th^j predetermined criterion of similarity, 
outputting an indicator indicating that a temporally predictive error 
concealment method should be used by a subsequent decoder. 

Consequently, the decoder is free to choose an appropriate type of error 
concealment method for a corrupted picture on the basis of the indicator. 
However the indicator does not specify' a particular algorithm and thus the 
decoder is not restricted to using a panic ular algorithm. 



P. 010^ 



30 



Preferably the error concealment Indies tor is updated when the measure of 
similarity does not meet the predeternined criterion. The indicator then 
indicates what may be viewed as a change in scene. 
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Thus the invention enables the decoder {o detect which pictures belong to the 



same scene and, on the basis of this i 
of error concealment method, if requirec 
indicator changes from one frame to 
applying a non-predictive error 
indicator is the same from one frame 



concealment 



incjication, to select an appropriate type 
, Thus, when the error concealment 
another, the decoder responds by 
method. If the error concealment 



temporally predictive error concealment method 



For example, at a scene change, the E;imilarity between the current picture 



(the first picture in a new scene) and the 
be low. The encoder will therefore upde 
the current picture. If the picture is cora 



to another, responds by applying a 



last picture of the previous scene will 
te the error concealment indicator for 
pted, the decoder detects the change 



In the error concealment indicator ard uses a non-temporally predictive 
concealment method. Additionally the :iecoder can determine which INTRA 
pictures are used to code a scene change and which ones are inserted into 
the video sequence for other reasons and can select an appropriate 
concealment algorithm on the basis of tK is information. 



Some video encoders code all frames (after the initial frame) as INTER 
pictures. If the first INTER picture arter a scene cut gets comjpted, a 
traditional decoder is likely to utilise tjemporal prediction in an attempt to 
conceal the errors introduced by loss of 
contents of two pictures from different 
picture may be unacceptably distorted. 



P. Oil 



an INTER picture. Consequently, the 
scenes are mixed and the concealed 
However a decoder according to the 



25 invention is able to detect on the basis of the error concealment indicator that 
it should use a non-temporal conceal Tient method instead of a temporal 



concealment method. Hence a picture 
error concealment. 



from a different scene is not used for 



30 



Compared with the solution introduced in MPEG-2, which requires additional 
motion rectors, the invention requires considerably fewer bits per frame. In 
addition, the invention operates with e xisting video compression standards 
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that do not allow motion vectors for INTR>^ macroblocks, as allowed in MPEG- 



The error concealment indicator may be included in respect of the whole 
picture or part of a picture. In the foimer case, typically the indicator is 
included in a picture header. In a preferred implementation of the invention, 
the video signal is encoded according tp the H.263 standard and the error 
concealment indicator is included in 
Information. In the latter case the error 
included in picture segment headers or 
picture. 



the Supplemental Enhancement 
concealment indicator may also be 
nacroblock headers of the encoded 



According to a second aspect of the invention there is provided a method of 
encoding a video signal representing £ sequence of pictures, the method 
comprising comparing a first picture with a second picture, calculating a 
measure of the similarity between the first and the second pictures, comparing 
the measure of similarity with a predotermined criterion of similarity and 
outputting an indicator in response to the measure of similarity wherein, when 
the measure of similarity does not moet the predetermined criterion, the 
indicator is updated and when the measure of similarity meets the 
predetermined criterion, the indicator is unchanged. 



representing 



encoded 



According to a third aspect of the invehtion 
decoding an encoded video signal 
method comprising receiving an 
picture to be decoded an indicator indiCc 
to be used in the decoding process and 
using a concealment method of the identified 



there is provided a method of 
a sequence of pictures, the 
video signal, identifying for each 
ting the type of concealment method 
decoding the encoded video signal 
type. 



P. 012 /g^ 
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According to a fourth aspect of the inv^sntion there is provided a method of 
error concealment comprising receiving an encoded video signal including an 
indicator indicating the type of concealment method to be used in the error 
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concealment process and concealinc 
appropriately. 



According to a fifth aspect of the invention there is provided a rnethod of 
decoding an encoded video signal repre senting a sequence of pictures, the 
method comprising receiving an encoded video signal, identifying for each 
picture to be decoded an indicator ref resenting the measure of similarity 
between a first picture and a second p cture and. when the indicator is the 
same as that of a previously received p cture, applying a temporal predictive 
error concealment method and, when th^ indicator is the different from that of 
a previously received picture, applying a 



the error in the video signal 



spatial error concealment method. 



According to a sixth aspect of the invent on there is provided a video encoder 
comprising an input for receiving a video signal representing a sequence of 
pictures, a calculator to calculate a mee sure of the similarity between a first 
and a second picture, and a comparator to compare the measure of similarity 
with a predetermined criterion of similarit/ and to output an indicator indicating 
the concealment method to be used by a subsequent decoder, the 
comparator being arranged to output an indicator indicating that a non- 
temporally predictive concealment method should be used when the measure 
of similarity does not meet the predstermined criterion, and, when the 
measure of similarity meets the predetermined criterion, to output an indicator 
indicating that a temporally predictive coicealment method should be used by 
a subsequent decoder. 



According to a seventh aspect of the 
encoder for encoding a video signal rep 



invention there is provided a video 
-esenting a sequence of pictures, the 



encoder comprising a comparator for cc mparing a first picture with a second 
picture, a processor for calculating a nreasure of the similarity between the 
first and the second pictures and comparing the measure of similarity with a 
predetermined criterion of similarity. th€ processor being arranged to output 
an indicator in response to the measure of similarity wherein, when the 



15-05-00 19:25 



+44 1252 865080 



P.M 



R-184 



Job-M3 



5-MAY. ' OO(MON) 18:29 NMP PATENTS UK 



FAX:+44 1252 865080 



10 



15 



20 



25 



30 



nc29788a.doc 



10 



measure of similarity doas not meet the 
is updated and when the measure of 
criterion, the indicator is unchanged. 



predetermined criterion, the indicator 
similarity meets the predetermined 



invention there is provided a video 



According to an eighth aspect of the 
decoder comprising an input for receiving an encoded video signal 
representing a sequence of pictures, a controller for identifying within the 
video signal for each picture to be decoded an indicator indicating the type of 
concealment method to be used in the decoding process, and decoding the 
encoded video signal using a concealment method as indicated by the 
indicator 



way of example only, with reference 



The invention will now be described, by 
to the accompanying drawings, in which: 

Figure 1 shows a multimedia mobile communications system; 
Figure 2 shows an example of the multimedia components of a multimedia 
terminal; 

Figure 3 shows an example of a video codec; 

Figure 4 shows the syntax of a bit stream as known according to H.263; 
Figure 5a shows an example of a bit stream output by an encoder according 
to a first implementation of the invention 
Figure 5b shows an example of a bit st 
to a second implementation of the inven 

Figure 6a shows an example of a bit stream output by an encoder according 
to a third implementation of the inventior ; 
Figure 6b shows an example of a bit stream output by an encoder according 
to a fourth implementation of the inventic^n; 
Figure 7 shows an example of a bit stream output by an encoder according to 
a fifth implementation of the invention. 



eam output by an encoder according 
ion; 



Figure 1 shows a typical multimedia mctbile 
multimedia mobile terminal 1 



communications system. A first 
communicates with a second multimedia mobile 
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terminal 2 via a radio link 3 to a mobile 
data is sent between the two terminals 1 , 

Figure 2 shows the typical multimedia 
terminal comprises a video codec 10, 
manager 30. a control manager 40, a 
modem 60 (if required). The video codec 
video capture device of the terminal (not 
signals for decoding from a remote term 



communications network 4. Control 
2 as well as the multimedia data. 

components of a terminal 1 . The 
an audio codec 20, a data protocol 
multiplexer/demultiplexer 50 and a 
10 receives signals for coding from a 
shown) (e.g. a camera) and receives 
nal 2 for display by the terminal 1 on 



10 a display 70, The audio codec 20 receives signals for coding from the 
microphone (not shown) of the terminal 1 and receive signals for decoding 
from a remote terminal 2 for reproduction by a speaker (not shown) of the 
terminal 1 . 

15 The control manager 40 controls the operation of the video codec 10, the 
audio codec 20 and the data protocol manager 30. However, since the 
invention is concerned with the operation of the video codec 10, no further 
discussion of the audio codec 20 and pre tocol manager 30 will be provided. 



er 



20 Figure 3 shows an example of a video 
The video codec comprises an encod 
The encoder part 100 comprises an inpu 
a camera or video source (not shownj) 
switches the encoder between an 

25 mode. 



sodec 10 according to the invention. 

part 100 and a decoder part 200. 
: 101 for receiving a video signal from 
of the terminal 1. A switch 102 
INTRA-mode of coding and an INTER- 



In INTRA-mode, the video signal from the input 101 is transfonmed into DCT 
co-efficients by a DCT transformer 103. The DCT coefficients are then 



passed to a quantiser 1 04 that quantises 
and the quantiser 104 are controlled by 

the video codec which also receives feedback control from the receiving 
terminal 2 by means of the control manatjer 40. 



the coefficients. Both the switch 102 
an encoding control manager 105 of 
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In INTER mode, the switch 102 is operated to accept from a subtractor 106 
the difference between the signal from he input 101 and a previous picture 
which is stored in a picture store 107. The difference data output from the 
subtractor 106 represents the prediction arror between the current picture and 
the previous picture stored in the picturo store 107. The data in the picture 
store 1 07 is generated by passing the dala output by the quantiser through an 



inverse DCT transform 109 to the 



Inverse quantiser 108 and applying an 
inverse-quantised data. The resulting data is added to the contents of the 
picture store 107 by adder 1 10. A motion estimator 1 1 1 may generate motion 
compensation data (motion vectors) frorr 
conventional manner. 



the data in the picture store 107 in a 



The video codec outputs the quantised [)CT coefficients 112a, the quantising 
index 112b (i.e. the details of the quamising used), an INTFWINTER flag 
1 12c to indicate the mode of coding perf ormed (I or P/B), a transmit flag 1 12d 
to indicate the number of the frame beinjj coded and the motion vectors 1 12e 
for the picture being coded. These are nr i 
50 together with other multimedia signals. 



The decoder part 200 of the video code 
120» an inverse DCT transformer 121, 
store 123 and a controller 124, The 
control signals demultiplexed from the 
demultiplexer 50. In practice the con 
controller 124 of the decoder may be the 



ultiplexed together by the multiplexer 



2 10 comprises an inverse quantiser 
motion compensator 122, a picture 
Controller 124 receives video codec 
encoded multimedia stream by the 
roller 105 of the encoder and the 
same processor. 



The operation of an encoder according to the invention will now be described. 
The video codec 10 receives a video signal to be encoded. The encoder 100 
of the video codec encodes the video signal by performing DCT 
transformation, quantisation and motion compensation. The decoded video 
data Is then output to the multiplexer 50 The multiplexer 50 multiplexes the 
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video data from the video codec 10 and 
well as other signals as appropriate) into 
outputs this multimedia signal to the rece 
required). 



control data from the control 40 (as 
a multimedia signal. The terminal 1 
ving tenminal 2 via the modem 60 (if 



The invention relates to the inclusion of an indicator in the video sequence 
syntax that indicates the type of conceblment method to be used by the 
decoder. This may be included in the picture layer of the bit stream or be 
provided as a separate layer. The indicator may enable identification of 
different scenes and thus the type of concealment method to be used. 

In a first embodiment of the invention, the error concealment indicator 



comprises a so-called scene identifier (S 



representation of each coded picture. The^ scene identifier is a value, which is 



the same for all of the pictures of the 
different scenes typically have different 



I), which is added to the bit-stream 



same scene. Pictures belonging to 
scene identifiers. Preferably, the 



value of SI is incremented each time it is updated. However, rt is envisaged 
that the scene identifier may take one of two values, with pictures from 
adjacent scenes having non-identical SI values e.g. the pictures of the odd- 
numbered scenes having SI set to O anq the pictures of the even-numbered 
scenes having SI set to 1 . 



The controller 105 of the encoder detects a scene cut by comparing the 
similarity between one frame and another. The controller 105 compares the 
pixel representation of the current picture; (received from input 102) with the 
pixel representation of the previous picture stored in the picture store 107. 
Each time the similarity between the two pictures is below a certain threshold, 
the controller detects a scene cut and un updated scene indicator 112f is 
output. If the similarity is above the threshold the scene indicator is not 
updated but is repeated. 
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The scene identifier does not necessf 
change in the scene identifier indicates 

sufficiently with respect to a previous pijjture that the current picture may be 
deemed to be a new scene. 



Other known methods for detecting seen 



rily identify an actual scene cut; a 
that the current picture has changed 



5 changes may be used. 



In a second embodiment of the invention, a so-called concealment method 
indicator (CMI) is added to the bit-stieam representation of each coded 
picture. The CMI indicates to a decoder the type of concealment method that 
the decoder should use if the associated picture is corrupted. The encoder 
decides which CMI to associate with th 
between a current picture and a previous 
pictures is below a certain threshold, the 
method indicator CMI=0 as output 11 2f 
non-temporal predictive concealment method should be used. If th© similarity 
is above the threshold a second concealment method indicator CM|=1 js 
output as output 11 2f. This indicates to 
concealment method should be used. 



3 picture by comparing the similarity 
one. If the similarity between the two 
controller outputs a first concealment 
This indicates to a decoder that a 



The measure of similarity may be 
correlation between the two pictures. 



An error concealment indicator may also be included in the picture segment 



25 header and/or the macroblock header to 



a decoder that a temporal predictive 



fomied by calculating the mathematical 



nfluence at this level the selection by 



a receiving decoder of an appropriate error concealment method. 



Considering the terminal 1 as receiving 
the operation of the video codec 10 will 
decoding role. The temninal 1 
transmitting terminal 2. The demultipl 
signal and passes the video data to the 



rece ves 



exer 



encoded video data from terminal 2, 
ow be described with reference to its 
a multimedia signal from the 
50 demultiplexes the multimedia 
video codec 10 and the control data 
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200 of the video codec decodes the 



to the control manager 40. The decoder 
encoded video data by inverse quantising, inverse DCT transforming and 
motion compensating the data. The coni roller 124 of the decoder checks the 
integrity of the received data and, if an vrror is detected, attempts to correct 
and conceal the error in a manner to pe described below. The decoded, 
corrected and concealed video data is 
display 70 of the receiving terminal 1 . 



Errors in video data may occur at the pic^ture 
or the macroblock level. Error checking 
these levels. 



Considering first a signal encoded acccrding to the invention, in which an 



then output for reproduction on a 



level, the picture segment level 
may be carried out at any or all of 



detected, the decoder examines the 
If the SI of the corrupted picture is 



indicator SI is included, when an error is 
SI of the received but corrupted picture, 
the same as that of the temporally-neighbouring correctly decoded picture(s)> 
the decoder applies a temporal predictipn (INTER) concealment algorithm. 
The decoder may be capable of carry 
concealment algorithms. The indicator S 

concealment algorithm that should be us^d but does not specify the algorithm 
to be used. 



di fe 



If the SI of the corrupted picture 
neighbouring correctly decoded picture(£ ) 
using a spatial (INTRA) concealment method 



Considering a signal encoded according 1o the invention, in which an indicator 
CM! is included, when an error is detected, the decoder examines the CMI of 
the received but corrupted picture. If the C^MI of the corrupted picture is CMI1, 
the decoder conceals the picture using c, spatial concealment method, if the 
CMI of the corrupted picture is CMI2. the decoder applies a temporal 
prediction concealment algorithm. Aga n the decoder may be capable of 



ng out various temporal prediction 
indicates to the decoder the type of 



rs from that of the temporally- 
, the decoder conceals the picture 
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carrying out various error concealmeit 
indicates to the decoder the type of con 
but does not specify the algorithm to be ilised 



An example of how the error concealm 
syntax of an encoded signal will now 
H.263 video coding standard. 



algorithms. The indicator CMI 
dealment method that should be used 



ant indicator may be included in the 
De addressed with reference to the 



Figure 4 shows the syntax of a bit strean as known according to H.263. (The 



following implementations describe the 
skilled person that the invention may aisc 
As mentioned already, the bit stream has 



segment layer, macroblock layer and block layer. The picture layer comprises 



a picture header followed by data for the 
by any optional end-of-sequence code ar 



PSC 
TR 



The prior art H.263 bit stream is formatte|d as shown in Figure 4. A descriptor 
for each part is given below: 

The picture start code (PSC) indicates the start of the picture 
The Temporal Reference (TR) is formed by incrementing its 
value in the temporally previous reference picture header by 
one plus the number of skipped or non-referenced pictures 
since the previously transnnitted one 

Amongst other things. PT/PE includes details of the picture 
coding type i.e. INTRA or IPJTER 

A codeword that indicates; the quantiser to be used for the 
picture until updated by an\ subsequent quantiser information 
A codeword that signals the use of optional continuous 
presence multipoint and video multiplex (CPM) mode 
Picture Sub-Bit stream Indicator - only present if CPM is set 
Present if the frame is a bi-directionally predicted frame (known 
as a PB-frame) 



PTYPE 

PQUANT 

CPM 

PSBI 
TRb 



30B format but it will be clear to a 
be implemented in the slice format.) 
four layers: the picture layer, picture 



Group of Blocks, eventually followed 
d stuffing bits. 
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17 

Present if a bi-directional fr^me 
This relates to extra insertion information and is set to "V to 



the following optional data fields 
and PEI are together known as 
Information, which is further 



Is the data for the group of 
A stuffing codeword provicUed 



Diocks for the current picture 

to attain byte alignment before 



10 EOS 



PSTUF 



A codeword indicating the 
picture 

A stuffing codeword to al 
picture start code PSC 



The structure as shown in Figure 4 does 
data field. PSBI is only present if indicated 
only present if PTYPE indicates use of a 
PLUSTYPE field is present and the usee 
These issues are addressed in more deta 



end of the data sequence for the 



ow for byte alignment of the next 



not include the optional PLUSTYPE 
by CPM. TRb and DBQUANT are 
-called PB frame mode (unless the 
of DBQUANT is indicated therein): 
I in the H.263 specification. 



so 



The following paragraphs outline possibib implementations of the bit-stream 
output by an encoder according to the invention. 

The concealment method indicator may be incorporated into a H.263 bit 
stream as follows. Figure 5a shows an example of a bit stream output by an 
encoder according to the first implementation of the invention. As shown in 
Figure 5a, the bit stream includes an additional codeword SI which is a 
codeword indicating the scene to which the picture belongs. This is inserted 
by the encoder according to the amotnt of change between successive 
pictures, as described above, A charge in identifier indicates that the 
similarity between the picture being coded and a reference picture is low. The 
decoder uses this information to select an error concealment method of the 
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indicated type. In this case, the type of (jrror concealment method to be used 
is not specifically indicated, but can be dotermined from the changes in SI. 



stream 



inve ntion 



Figure 5b shows an example of a bit 
to the second implementation of the 
stream may include an additional codeword 
indicating the type of concealment methcjd 
inserted by the encoder according 
successive pictures, as described above. 



tc 



output by an encoder according 
. As shown in Figure 5b, the bit 
CMI which is a codeword 
to be used by the decoder. This is 
the amount of change between 



Alternatively, the SI or CMI may he included in the Supplemental 
Enhancement Infomiation PSUPP (see Annex L of H.263 and Figure 4) The 
supplemental information may be present in the bit stream even though the 
decoder may not be capable of providing the enhanced capability to use it. or 
even to properly interpret it. Simply discarding the supplemental infonnation 
is allowable by decoders unless a recuirement to provide the requested 
capability has been negotiated by externa I means. 



If PEI is set to "1 \ then 9 bits follow 
then another PEI bit to indicate if a furthe 



consisting of 8 bits of data (PSUPP) and 
9 bits follow and so on. 



The PSUPP data consists of a 4-bit func:ion type indication FTYPE, followed 
by a 4-bit parameter data size specification DSIZE followed by DSIZE octets 
of function parameter data, optionally followed by another FTYPE and so on. 
It is known to use this PSUPP codeword to signal various situations such as: 
to indicate a full-picture or partial-picture ireeza or f reeze-release request with 



or without resizing; to tag particular picti 
the video stream for external use; or to 
video compositing. 



res or sequences of pictures within 
convey chroma key information for 
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To implement the invention using 
Infomnation. a further FTYPE is defined 
example. FTYPE 15 may be used for this 



the Supplemental Enhancement 
as "Scene Identifier" or "CMl". For 
purpose. 

This is illustrated in Figures 6a and 6bL Figure 6a illustrates the example 
where a parameter SI is included in the SEI of the picture header. The 
FTYPE is defined as Scene Identifier SI. The DSIZE specifies the size of the 
parameter and the following octet is the parameter data i.e. the value of SI. 
From this value a receiving decoder c:an determine whether a corrupted 
picture is from the same "scene" as a p revious one and so select the most 
appropriate type of concealment method. Figure 6b illustrates the example 
where a parameter CMl is included in the SEI of the picture header. The 
FTYPE is defined as Concealment Method Indicator CMl. The DSIZE 
specifies the size of the parameter and the following octet is the parameter 
data i.e. the value of CMl. From this valte a receiving decoder can determine 
the most appropriate type of concealmen : method for a corrupted picture. 

Alternatively, the information may be contained in the additional Supplemental 
Enhancement Information as specified in a "Draft of new Annex W: Additional 
Supplementary Enhancement Information Specification" P. Ning and S 
Wenger, ITU-T Study Group 16 Questioiti 15 Document Q1 5-1-58, November 
1999. 



In this draft proposal, FTYPE 14 is defined 
FTYPE is set, the picture message funct 
more octets representing message data, 
is a message header with the structure 
and MTYPE. DSIZE is equal to the nuitnber 
corresponding to a picture message function 
header. 



as "Picture Message". When this 
on indicates the presence of one or 
The first octet of the message data 
shown In Figure 7 i.e. CONT, EBIT 
of octets in the message data 
, including the first octet message 
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1, indicates that the message data 



The continuation field CONT, if equal to 
associated with the picture message is p art of the same logical message as 
the message data associated with the re^ct picture message function. The 
End Bit Position field EBIT specifies the number of least significant bits that 
shall be ignored in the last message octe t. Further details of these fields can 
be found in the draft of Annex W, referrec to above 



essage. Various types of message 
According to the invention one type 



The field MT/PE indicates the type of rr 
are suggested in the draft of Annex W, 
e.g. MTYPE 9 is defined as Error Concea lment Type. The value of the scene 
identifier or CMI is defined in the octet following the message header For the 
scene identifier the value is the same in all pictures of the same scene. 
Pictures belonging to different scenes hs ve different scene identifier values. 
The decoder uses the SI or CMI to detamiine the type of error concealment to 
be used. 



received 



The message may also indicate the type 
used for particular specified rectangular 
part of the area is not correctly 
concealment type messages for one picti 
type for a non-overiapping rectangular 
do not cover some areas of the 
concealment for those areas. Preferably, 
type that corresponds to the picture 
concealment method for an INTER- 
concealment method for an INTRA-frame. 



areas 



frane 



of error concealment that should be 
of the current picture if at least 
There may be multiple error 
re each specifying the concealment 
. Advantageously if the messages 
a decoder uses any error 
the decoder uses the concealment 
type i.e. a temporal predictive 
and a non-temporal predictive 



area 



pic:ture 



For each error concealment type 



A specific example will now be given, 
message, DSIZE shall be 6, CONT shall be 0. and EBIT shall be 0. If the first 
data byte is equal to one (0000 0001), this indicates to a decoder that 
transmission errors are to be concealed u sing spatial concealment algorithms 
only. If the first data byte is equal to tv^o (0000 0O10). this indicates to a 
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to ba concealed using temporal 



decoder that transmission errors are 

prediction. The following four PSUPF^ octets contain the horizontal and 
vertical location of the upper left come" of the specified rectangular region 
within the picture within which errors are to be concealed, and the width and 
height of the rectangular region, respectively, using eight bits each and 
expressed in units of 16 pixels (of lumins nce picture). For example, an entire 
Quarter Common Intermediate Format (CJCIF) picture (which has a luminance 
picture of 176 X 144 pixels) is specified by the four parameters (O, 0, 1 1 , 9). 



10 For picture formats having a width and 



leight that is not divisible by 16, the 



specified area may extend to the next larger size that would be divisible by 1 6. 
For example, an entire image having size of 160 x 120 pixels is specified by 
the four parameters (0, 0, 10, 8). Pre erably, the specified area does not 



cross picture boundaries, and preferably 



does not overlap with other specified 



1 5 error concealment areas of the same picture. 



The error concealment indicator may also be incorporated into other video 
coding protocols. For example MPEG-4^ defines so-called user data, which 
can contain any binary data and is not necessarily associated with a picture. 
The error concealment identifier may be added to these fields. 



The invention is not intended to be liriited to the video coding protocols 
discussed above: these are intended to be merely exemplary. The invention 
is applicable to any video coding protocol in which a choice of error 
concealment methods is available. Tihe addition of the information as 
discussed above allows a receiving decoder to determine which would be the 
best type of error concealment method to use 
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CLAIMS 



1. A method of encoding a video signal representing a sequence of 
pictures, the method comprising comparing a first picture with a second 
picture, calculating a measure of the ssimilarity between the first and the 
second pictures, comparing the measure of similarity with a predetemrilned 
criterion of similarity and, when the measure of similarity does not meet the 
predetermined criterion of similarity, out cutting an indicator indicating that a 
non-temporally predictive error concealment method should be used by a 
subsequent decoder and, when the measure of similarity meets the 
predetermined criterion of similarity, outputting an indicator indicating that a 
temporally predictive error concealment method should be used by a 
subsequent decoder. 



2. A method according to claim 1 , 
the measure of similarity does not 
similarity. 



wnerein the indicator is updated when 
mieet the predetermined criterion of 



3. A method according to claim 1 or g wherein the indicator is included in 
a picture header. 

4. A method according to claim 3 v^herein the video signal is encoded 
according to the H.263 standard ancj the indicator is included in the 
Supplemental Enhancement Information 



5. A method of encoding a video 
pictures, the method comprising 
picture, calculating a measure of the 
second pictures, comparing the measu 
criterion of similarity and outputting an inc 
similarity wherein, when the measure 



signal representing a sequence of 
companng a first picture with a second 
4imilarity between the first and the 
of similarity with a predetermined 
icator in response to the measure of 
of similarity does not meet the 
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predetermined criterion, the indicator is 
similarity meets the predetermined criteri(t)n 



encoded 



6. A method of decoding an 
sequence of pictures, the method compnsi 
signal, identifying for each picture to be 
type of concealment method to be 
decoding the encoded video signal usjing 
identified type. 



used 



7. A method of error concealment 
signal including an indicator Indicating 
used in the error concealment process 
signal appropriately. 



th2 



8. A method of decoding an encbded video signal representing a 
sequence of pictures, the method comorising receiving an encoded video 
signal, identifying for each picture to be decoded an indicator representing the 
measure of similarity between a first picture and a second picture and, when 
the indicator is the same as that of a previously received picture, applying a 
temporal predictive error concealment method and, when the indicator is the 
different from that of a previously recei\|6d picture, applying a spatial error 
concealment method. 



updated and when the measure of 
, the indicator is unchanged. 



video signal representing a 
ng receiving an encoded video 
decoded an indicator indicating the 
in the decoding process and 
a concealment method of the 



copnpnsing receiving an encoded video 
type of concealment method to be 
concealing the error in the video 



and 



9. A video encoder comprising an 
representing a sequence of pictures, a 
the similarity between a first and a 
compare the measure of similarity with a 
and to output an indicator indicating the 
subsequent decoder, the comparator be 
indicating that a non-temporally predicti>^e 
used when the measure of similarity 



input for receiving a video signal 
dalcu later to calculate a measure of 
seiond picture, and a comparator to 
predetermined criterion of similarity 
concealment method to be used by a 
ng arranged to output an indicator 
concealment method should be 
does not meet the predetermined 
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criterion, and. when the measure of 
criterion, to output an indicator i 
concealment method should be used by 



similarity meets the predetermined 
ndiqating that a temporally predictive 
subsequent decoder. 



10. A video encoder for encoding a v deo signal representing a sequence 
of pictures, the encoder comprising a comparator for comparing a first picture 
with a second picture, a processor for calculating a measure of the similarity 
between the first and the second pictures and comparing the measure of 
similarity with a predetermined criterion of similarity, the processor being 
arranged to output an indicator in response to the measure of similarity 
wherein, when the measure of similarity does not meet the predetermined 
criterion, the indicator is updated and wnen the measure of similarity meets 
the predetermined criterion, the indicator is unchanged, 

11, A video decoder comprising an input for receiving an encoded video 
signal representing a sequence of pictures, a controller for identifying within 
the video signal for each picture to be decoded an indicator indicating the type 
of concealment method to be used in the 
encoded video signal using a concea 
indicator. 



12. A portable radio communications device including an encoder 
according to claim 9 or 1 0 and/or a decoc er according to claim 10 or 1 1 . 



decoding process, and decoding the 
ment method as indicated by the 
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ABSTRACT 
VIDEO OODINO 





10 



A method of encoding a video signal reprssentirig a sequence of pictures, the 
method comprising comparing a first pictt re with a second picture, calculating 
a measure of the similarity between the first and the second pictures, 
comparing the measure of similarity with ci predetermined criterion of similarity 
and, when the measure of similarity does loX meet the predetermined criterion 
of similarity, outputting an indicator indicating that a non-temporaljy predictive 
error concealment method should be used by a subsequent decoder and, 
when the measure of similarity meets the predetermined criterion of similarity, 



outputting an indicator Indicating th£ 
concealment method should be used by a 



15 



t a temporally predictive error 
subsequent decoder. 
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